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HPA - Concept 
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Specify the target for the load 
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Target 


VPA - Concept 


* Free the user from the necessery of 
setting up to date resource limits 
and requests. 


$CPU, 166b 


* Will set resources limits and requests 
according to the pod's actual usage. 
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Supports scale-down and scale-up 


* Will most commonly be used for a 
deployment object 
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Cluster Autoscaler 
CA 
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steps to add new node(s) 
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Cluster Autoscaler 
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Cluster Autoscaler scale-up 
HEN — 


Pending 
pods | | | a ^g ME 
Cluster Autoscaler | 
EE ee Uu E E: 


! @ mcum 
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10 sec New nodes us 
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Auto scaling group — FHH — . Noce 
Expander 8vCPUs 32 GB Spot E 1vcPU ! 


Cluster AutoScaler RunOnce: 
° Reconciliation and filtering 
* Scale up (simulation and expander logic) 


* Scale down and filtering 
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Karpenter Cluster 
Autoscaler 
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How Karpenter provisions nodes on AWS 


consolidates instance orchestration responsibilities within a single system 
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Karpenter scale-up 


LL 
Karpenter — N — — 


Default: all H HH HH m EE 


instance types, 
10 sec excluding metal 


instanceTypes: 
[m5.large, m5.2xlarge, ...] 


Provisioning and scheduling decisions 
* Early binding to provisioned nodes vs. placeholder instances 
° Remove scheduler version dependency 
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Target 
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Karpenter scale-in 


Karpenter 


ttISecondsAfterEmpty: H HH Em FH 225 | m A be 
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seconds the controller will wait 
! B] 1vCPU 


10 sec before attempting to delete a 
node, measured from when the — a drcum 0 ease ee 
É request 
node is detected to be empty 


Terminations 
° Remove underutilized nodes (empty nodes) 
* Node TTL 

@ ° Consolidation 
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Karpenter Consolidation 


Karpenter simulates all pods being evicted from a candidate spec: 


node E 
consolidation: 
enabled: true 


Reduce the overall cost in two ways: 


*Node Deletion - A node is eligible for deletion if all of its 


pods can run on free capacity of other nodes in the cluster. 


*Node Replacement - A node can be replaced if all of its 


pods can run on a combination of free capacity of other 


nodes in the cluster and a single cheaper replacement nofgrotecting an Application 


with a 
PodDisruptionBug 
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Compute provisioning with Provisioner CRD 


* Provisioner - Custom Resource to ran 
provision nodes with a set of optional ide 
attributes (Taints, Labels, Requirements, "labels 
TTL) 1 SA 30 
- TN ttlSecondsUntilExpired: 2592000 
* Asingle provisioner can manage os 
compute for multiple teams and - key: kerpenter.sh/capacity-type 
WO rkloads ee ["spot“,"on-demand”] 
^ - key: "topology.kubernetes.io/zone" 
* Create a default provisioner (named E 
“default”) for common scenarios xr RM 
* Multiple provisioners for isolating Aa 
compute for different needs Js 
cpu: 1000 
provider: 


securityGroupSelector: 
karpenter.sh/discovery: $ (CLUSTER NAME) 
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Compute flexibility - Purchase Options and CPUs 


Purchase options — 
*Default is on-demand requirements: 
*Configure on-demand and Spot purchase - key: karpenter.sh/capacity-type 
ns operator: In 
*When on-demand and Spot are configured - values: ["spot", "on-demand"] 
Spot prioritized 
*Provisions on-demand when Spot constrained 
Spec: 
CPU architecture requirements: 
*Default is x86 instances only (amd64) - key: node.kubernetes.io/arch 
*Diversify across x86 and ARM architecture operator: In 
instances values: ["arm64", "amd64"] 
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Compute flexibility - Instance types and AZS 


Instance type 


Spec: 
*Defaults to all EC2 instance types : | 
excluding metal and GPU Kote) Me eM. Sm 
m - key: node.kubernetes.io/instance- 
Only restrict instance types if required 11 275 
instance diversification across YP 
ee operator: In 
© values: ["m5.large", "m5.2xlarge"] 
Families 
e Generations 


> we spec: 
requirements: 
Availability Zone - key: topology.kubernetes.io/zone 
* Defaults to all AZs a In 1.28 ^ E 
* Only restrict AZs if required SINA 323) Zn 
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Taints 


Startup Taints 


startupTaints: 
e Temporary nodes start with the taint - key: example.com/another-taint 
*DaemonSet will delete it (networking) | 
effect: NoSchedule 


l taints: 
Taints iss 


* Prevent pods from scheduling t d example.com/special- 
ain 


effect: NoSchedule 
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Scheduling 


Node Selector 


Ask for a node that matches selected key-value | nodeSelector: 

pairs topology.kubernetes.io/zone: us- 

west-2a 
karpenter.sh/capacity-type: spot 


*well-known labels or custom labels 


affinity: 
Node Affinity nodeAffinity: 
* requiredDuringSchedulinglgnoredDuringExecuti requiredDuringSchedulinglgnoredDuringExecution: 
on: - hard rule that must be met. nodeSelectorTerms: 
* preferredDuringSchedulinglgnoredDuringExecu - matchExpressions: 
tion: - preference, pod can run on a node - key: "topology.kubernetes.io/zone" 
where it is not guaranteed. operator: "In" 
values: ["us-west-2a, us-west-2b"] 
- key: "topology.kubernetes.io/zone" 
operator: "Notln" 
values: ["us-west-2b"] 
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Custom User Data and AMI 


apiVersion: karpenter.k8s.aws/vlalphal 
kind: AWSNodeTemplate 


metadata: 
name: bottlerocket-example 
spec: 
apiVersion: karpenter.sh/vlalpha5 amiFamily: Bottlerocket 
kind: Provisioner instanceProfile: MyInstanceProfile 
metadata: subnetSelector: 
name: default karpenter.sh/discovery: my-cluster 
spec: securityGroupSelector: 
providerRef: karpenter.sh/discovery: my-cluster 
name: bottlerocket-example userData: | 
[settings.kubernetes] 


kube-api-gps = 30 

[settings.kubernetes.eviction-hard] 

"memory.available" = "20%" 
amiSelector: 

karpenter.sh/discovery: my-cluster 
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Kubelet Configuration 


Kubelet configuration 
*Karpenter provides the ability to specify a few additional Kubelet args. 
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kubeletConfiguration: 
clusterDNS: ["10.0.1.100"] 
containerRuntime: containerd 
systemReserved: 
cpu: 1 
memory: 2G! 
ephemeral-storage: 5G! 
kubeReserved: 


evictionHard: 


Control Pod Density 


Networking Limitations 
*Number of networking interfaces (ENIS) 
Number of IP addresses that can be assigned to each ENI 


kubeletConfiguration: 
podsPerCore: 2 
maxPods: 20 


Static Pod Density 


Dynamic Pod Density 


Limit Pod Density 
* Topology Spread 
* Restrict Instance Types 
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